Text Retrieval using Linear Algebra

Text retrieval is an important area of research. As information and methods of its storage have proliferated, the need to have efficient methods of locating subsets of this information has increased as well. A widely-researched text searching method involves modeling a text collection in a term-by-document matrix, and evaluating the documents’ relevance to a query with simple linear algebra. This document is an abstract of research performed throughout 2001 and 2002, and presents one such system, developed to search the bsu-cs webserver. In this method, a matrix A is constructed such that the frequency of each term i in each document j, is stored at Ai,j . To build this matrix for bsu-cs, a simple index-creator was programmed. Beginning with a list of known Web pages, the script analyzed each document encountered. Word frequencies were recorded, and links to other bsu-cs Web pages were extracted. Those links were both added to the page list, and tabulated as possible indicators of document importance (more links to page x implies that page x is more important). Note that common words such as “the” and “how” have little intrinsic meaning [1]. Therefore, a list of “stop words” was created. The members of this list were excluded by the matrix-forming code. After several hours of parsing, the indexcreator, in conjunction with a script to organize each page’s raw data, produced a 26257 by 3557, sparse matrix. If a user supplies a list of key words, the relevance of those words to each of the documents in the term-by-document matrix A = [~a1,~a2, . . . ,~an] can be determined. A common measure of such is the cosine of the angle between the query vector, ~q and each document (column) vector, ~aj [3]. If the query is represented by a vector ~q and the term-by-document matrix as A, the cosine of the angle between ~q and a document ~aj is found by: